Investigating the Soar-RL Implementation of the MAXQ Method for Hierarchical Reinforcement Learning
نویسنده
چکیده
Discussed in greater detail below, Soar-RL is the integration of the reinforcement learning method of machine learning into Soar, a generalized architecture. The MAXQ method for hierarchical reinforcement learning [1] greatly influenced the design for the hierarchical reinforcement learning components of Soar-RL [2]. In its pre-release form, it is prudent to question the merits of this union: what, conceptually and computationally, have we gained and lost by implementing a highly optimized algorithm in a general architecture? Intuitively, abstracting a problem implementation carries a computational cost, in the form of increased time/space requirements. Additionally, when moving from the low-level control of a custom solution to an architectural paradigm, we may suffer from reduced ability to direct program behavior. However, modern programs are not typically written in assembler: abstraction has its benefits. Most pertinent, with abstraction comes the ability to quickly generate, tune, and explore relatively large numbers of problem instances. We dedicate a large portion of this project effort to comparing these tradeoffs in context of a complex, hierarchical reinforcement learning domain.
منابع مشابه
State Abstraction in MAXQ Hierarchical Reinforcement Learning
Many researchers have explored methods for hierarchical reinforcement learning (RL) with temporal abstractions, in which abstract actions are defined that can perform many primitive actions before terminating. However, little is known about learning with state abstractions, in which aspects of the state space are ignored. In previous work, we developed the MAXQ method for hierarchical RL. In th...
متن کاملContinuous-Time Hierarchical Reinforcement Learning
Hierarchical reinforcement learning (RL) is a general framework which studies how to exploit the structure of actions and tasks to accelerate policy learning in large domains. Prior work in hierarchical RL, such as the MAXQ method, has been limited to the discrete-time discounted reward semiMarkov decision process (SMDP) model. This paper generalizes the MAXQ method to continuous-time discounte...
متن کاملAn Evolutionary Approach to Automatic Construction of the Structure in Hierarchical Reinforcement Learning
Because the learning time is exponential in the size of the state space, a hierarchical learning structure is often introduced into reinforcement learning (RL) to handle large scale problems. However, a limitation to the use of hierarchical RL algorithms is that the learning structure, representing the strategy for solving a task, has to be given in advance by the designer. This thesis presents...
متن کاملPotential Based Reward Shaping for Hierarchical Reinforcement Learning
Hierarchical Reinforcement Learning (HRL) outperforms many ‘flat’ Reinforcement Learning (RL) algorithms in some application domains. However, HRL may need longer time to obtain the optimal policy because of its large action space. Potential Based Reward Shaping (PBRS) has been widely used to incorporate heuristics into flat RL algorithms so as to reduce their exploration. In this paper, we inv...
متن کاملThe MAXQ Method for Hierarchical Reinforcement Learning
This paper presents a new approach to hierarchical reinforcement learning based on the MAXQ decomposition of the value function. The MAXQ decomposition has both a procedural semantics—as a subroutine hierarchy—and a declarative semantics—as a representation of the value function of a hierarchical policy. MAXQ unifies and extends previous work on hierarchical reinforcement learning by Singh, Kae...
متن کامل